A regularized stochastic configuration network based on weighted mean of vectors for regression

The stochastic configuration network (SCN) randomly configures the input weights and biases of hidden layers under a set of inequality constraints to guarantee its universal approximation property. The SCN has demonstrated great potential for fast and efficient data modeling. However, the prediction accuracy and convergence rate of SCN are frequently impacted by the parameter settings of the model. The weighted mean of vectors (INFO) is an innovative swarm intelligence optimization algorithm, with an optimization procedure consisting of three phases: updating rule, vector combining, and a local search. This article aimed at establishing a new regularized SCN based on the weighted mean of vectors (RSCN-INFO) to optimize its parameter selection and network structure. The regularization term that combines the ridge method with the residual error feedback was introduced into the objective function in order to dynamically adjust the training parameters. Meanwhile, INFO was employed to automatically explore an appropriate four-dimensional parameter vector for RSCN. The selected parameters may lead to a compact network architecture with a faster reduction of the network residual error. Simulation results over some benchmark datasets demonstrated that the proposed RSCN-INFO showed superior performance with respect to parameter setting, fast convergence, and network compactness compared with other contrast algorithms.


INTRODUCTION
Neural networks have shown superiority over data modeling because of their powerful representation learning ability to learn patterns with multiple levels of abstraction that make sense to the data (Bengio, Courville & Vincent, 2013). However, the gradient-based iterative training process of neural networks is time-consuming and computationally intensive (Wang & Li, 2017b). Feed-forward neural networks (FNNs) with random parameters have drawn widespread attention due to their faster training speed and lower computational cost (Scardapane & Wang, 2017). Igelnik & Pao (1995) found that any continuous function can be approximated by a random vector functional link (RVFL) network with probability one under appropriate parameters. The hidden parameters of RVFL were assigned randomly in a preset scope and the output weights were calculated based on the least squares method (Cao et al., 2021). However, determining the preset scope of randomized learning models is challenging, and the widely used scope of random parameters (e.g., [−1,1]) is not always feasible (Li & Wang, 2017).
To resolve the infeasibility of using RVFL networks for data modeling with a fixed scope (i.e., [−1,1]), Wang & Li (2017b) proposed a novel randomized learning framework, termed SCN. The hidden parameters (input weights and biases) of SCN are randomly assigned under a supervisory mechanism and adaptively select their ranges, which indicate prominent merits on human intervention of network structure, range adaptation of hidden layer parameters, and sound generalization (Dai et al., 2019a).
Many efforts have been made to enhance the performance of SCN since it was developed in 2017. SCN with kernel density estimation (RSC-KDE) and maximum correntropy criterion (RSC-MCC) were presented to weaken the negative influences of noise and outliers, respectively, on modeling performance (Wang & Li, 2017a;Li, Huang & Wang, 2019). Zhu et al. (2019) delved deeper into the inequality constraint used in SCN and presented two new inequalities to increase the probability of satisfying the constraint condition. As deep neural networks (DNNs) with multiple levels of feature extraction can learn more abstract representations of the data, a deep version of SCN (DeepSCN) with multi-hidden layer network structure was proposed by Wang & Li (2018). For image data analysis with matrix inputs, a two-dimensional version of SCN (2DSCN) was proposed by Li & Wang (2019). For SCN ensembles, Wang & Cui (2017) adopted the negative correlation learning (NCL) ensemble learning technique to reduce the covariance among the base SCN for large-scale data analysis. Huang, Li & Wang (2021) designed a novel indicator that contained some key factors to explore appropriate base learner models from a set of SCN to generate an effective ensemble model.  developed a parallel SCN (PSCN) by introducing the beetle antennae search (BAS) optimization algorithm and fuzzy evidence theory for large-scale data regression. For finding the optimal parameter settings,  utilized the chaotic sparrow search algorithm to optimize the contractive factor r in the inequality and the scale factor λ of random parameters to enhance the effectiveness of SCN. In addition, various extensions of SCN were applied to data modeling in real-world applications, such as molten iron quality (MIQ) modeling in blast furnace ironmaking (BFI) (Xie & Zhou, 2020), particle size estimation of hematite grinding process (Dai et al., 2019b), traffic state prediction across geo-distributed data centers of the China Southern Power Grid (CSG) , and prediction of asphaltene and total nitrogen in crude oil (Lu & Ding, 2019;Lu et al., 2020).
SCN starts with a small-sized network structure and gradually adds new hidden nodes into the network until the residual error of SCN is smaller than the tolerance threshold.
With the increasing number of hidden nodes, the constructive SCN model is prone to overfitting and thus poor performance . Meanwhile, the performance of SCN is frequently impacted by the parameter settings of the model, such as λ (the scale factor of weights and bias) and r (the contractive factor in the inequality). Seeking better model parameters is vital for SCN. It is well known that the L 2 regularization technique, which adds the "squared magnitude" of the coefficient to the loss function, can prevent the problem of overfitting effectively. In the famous Residual Network (ResNet), He et al. (2016) let the stacked nonlinear layers fit a residual mapping of FðxÞ :¼ HðxÞ À x. Inspired by the idea of residual learning in ResNet, we used the current network residual error feedback to dynamically adjust the parameters of SCN.
Therefore, the objective of this study was to automatically obtain better parameters for SCN and get a more compact architecture. First, the L 2 regularization item combined with network residual error was introduced to improve the generalization performance of SCN. In addition, a regularized SCN based on INFO was developed to optimize the parameter selection of SCN. INFO is a relatively new swarm intelligence optimization method published in 2022. Updating rule, vector combining, and a local search were the three core phases of INFO (Ahmadianfar et al., 2022). It is a promising tool for the parameter optimization of the regularized stochastic configuration network (RSCN). To summarize, the key contributions of this article are as follows: Introduce the regularization term that combines the ridge method with the network residual error into the objective function to dynamically adjust the training parameters of SCN. Optimize the scope setting of the input weights and biases λ, contractive factor r in the inequality, regularization coefficient g, and positive scale factor c of feedback residual error of RSCN by INFO, which in turn achieves a better RSCN model with respect to fast convergence and structure compactness. Illustrate the merits of RSCN-INFO on one function approximation and three benchmark regression datasets. The evaluation results justify the effectiveness of the proposed RSCN-INFO.

PRELIMINARIES
This section briefly reviews the classical SCN framework and the newer INFO algorithm.

SCN
SCN is a novel randomized incremental learner framework with a supervisory mechanism. The universal approximation property of SCN is guaranteed by its innovative inequality constraint. The network structure of SCN is depicted in Fig. 1.
Let À :¼ fg 1 ; g 2 ; g 3 ; . . .g be a set of real-valued functions, span(À) denotes a function space spanned by À, and L 2 ðDÞ represents the space of all Lebesgue measurable functions the output of a single layer feed-forward network with L À 1 hidden nodes, where w j and b j denote the parameters of the jth hidden node, and g j ðÁÞ is the activation function of the jth hidden node. The inner product of h ¼ ½h 1 ; h 2 ; . . . ; h m : R d ! R m and f is: Given a training dataset ðx i ; t i Þ N i¼1 , x i ¼ ½x i;1 ; . . . ; x i;d 2 R d and t i ¼ ½t i;1 ; . . . ; t i;m 2 R m , suppose that an established SCN model contains L À 1 hidden nodes. The network residual error is: . . . (2) where f is the given target function, and f LÀ1 represents the output of the network with L À 1 hidden nodes. The residual error of SCN gradually decreased as the number of hidden neurons increased. If the value of E LÀ1 was larger than the tolerance threshold s, SCN added a new hidden node into the network until E L was smaller than s. The parameters of the added hidden node were assigned randomly under a set of inequalities.
, E LÀ1;q ; g L . 2 ! b 2 g ð1 À r À l L Þjje LÀ1;q jj 2 ; q ¼ 1; 2; …; m; where , E LÀ1;q ; g L . is the inner product of E LÀ1;q and g L , fl L g represents a nonnegative real number sequence with lim L!þ1 l L ¼ 0, and l L ð1 À rÞ. g indicates a non-linear activation function.
. r determines the strictness of the inequality constraint, and 0 , r , 1. The output weights were evaluated by: where H y L represents the Moore-Penrose inverse of H L and H L is the output matrix of the hidden layer. Readers may refer to Wang & Li (2017b) for more details on the SCN framework and associated algorithms.

INFO
INFO is a new population-based optimization algorithm that employs updating rule, vector combining, and local search to move the population's position in D dimensional search domains. Given a population with N P vectors, X g l;j ¼ fx g l;1 ; x g l;2 ; …; x g l;D g; l ¼ 1; 2; …; N P .
Updating rule INFO randomly selected three differential vectors (a1 6 ¼ a2 6 ¼ a3) to calculate the weighted mean of vectors. To increase the diversity of the population, the best, better, and worst solutions were employed to define the MeanRule (mean-based rule). where where x ¼ f ðx ws Þ: The weighted mean of vectors was used to generate two new vectors.
where f ðxÞ was defined as the objective function, three different integers (a1; a2; a3) were randomly chosen from ½1; N P , z1 g l and z2 g l were two new vectors, and r was the scaling rate of a vector.

Vector combining
The two new vectors z1 g l and z2 g l were combined with vector x g l to generate a new vector l g l .
where l g l was the composite vector of the gth generation.

Local search
The local search operator used the global position (x g best ) and the MeanRule to help INFO convergence to global optima.
in which where rand and f were two random values within [0,1] and (0,1) respectively. The random value v 1 and v 2 increased the best position's influence on the vector. INFO updated the best vector (x best ) and returned x g best as the final solution. For more details about the INFO algorithm, refer to Ahmadianfar et al. (2022).

RSCN-INFO RSCN
Given a training dataset ðx i ; t i Þ N i¼1 , the objective function of the SCN with L 2 -norm penalty term could be expressed as: where hðx i Þ stands for the hidden output for the input x i , g is a non-negative real number, and the regularization coefficient g balances the residual error À P N i¼1 E 2 i Á and norm of the output weights (jjbjj 2 ). SCN added new hidden neuron b L ; g L ; incrementally leading to f L ¼ f LÀ1 þ b L g L . After dynamically adjusting the output weights during the training process, the current residual error was added to the L 2 regularization term. After adding the Lth new hidden node into an established SCN model with L À 1 hidden nodes, a new objective function was introduced.
where c is the positive scale factor of feedback residual error. The derivative of function Eq. (22) with respect to b L is: Letting Eq. (23) be equal to 0, The output weights of the L-th hidden node was obtained by: Theorem 1. Assume that spanðÀÞ is dense in L 2 space. Given 0 , r , 1, 0 , g, 0 , c, and a nonnegative real number sequence l L , with l L ! 1 À r and lim L!þ1 l L ¼ 0. 8g 2 À, 0 , jjgjj , b g for some b g 2 R þ . For L ¼ 1; 2; … and q ¼ 1; 2; …; m, the random basis function g L is generated by Eq. (25), and the output weights of the Lth hidden neuron are obtained by Eq. (26). Then, the SCN has lim L!þ1 jjf À f L jj ¼ 0.
Proof. First, the monotonically decreasing property of jjE L jj will be proved.
The monotonically decreasing property of jjE L jj has been proven. From Eqs. (25)- (27): According to Eq. (25): jjE L jj 2 À ðr þ l L ÞjjE LÀ1 jj 2 0: Therefore: Theorem 1 has given lim L!þ1 l L ¼ 0, which means lim L!þ1 l L jjE LÀ1 jj 2 ¼ 0. Based on Eq. (30), lim L!þ1 jjE L jj 2 ¼ 0. Therefore, lim L!þ1 jjE L jj ¼ 0. Remark 1. In theorem 1, the output weights are evaluated by Eq. (26) and kept fixed. This may cause a slow convergence rate. To cope with this problem, the output weights of all hidden neurons are updated by the least squares method after the new hidden node has been added. Let ½b Ã Hadamard product (element-wise multiplication) and E The output weights are calculated by: The output weights are recalculated in accordance with Eq. (31) as the newly added hidden neuron is generated to satisfy Eq. (25). The inequality constraint guarantees the universal approximation capability of RSCN. The process of proof is similar to theorem 1, so the detailed proof procedure is omitted. Remark 2. In Eq. (22), the residual error is added into the regularization term. The reason why jjg L jj 2 c is used instead of c is that the value of the residual error is equal to the output of training samples before adding hidden neurons into the network (E 0 ¼ T). So the residual error is relatively larger at the beginning of the construction process. It gradually decreases as the constructive process proceeds. Meanwhile, due to the randomness of SCN, g L is randomly generated under a set of inequality constraints. The scale factor jjg L jj 2 c makes it possible for the feedback residual error to adjust dynamically in pace with the change of the hidden output (g L ).

RSCN-INFO algorithm
INFO is a very competitive new optimization algorithm. In this section, INFO is applied to optimize the parameter λ, the contractive factor r, the regularization coefficient g, and the positive scale factor c for RSCN. The widely-used root mean square error (RMSE) is employed as the fitness function.

RMSE
For convenience's sake, n L;q ; q ¼ 1; 2; …; m is defined to describe the algorithm. The pseudo-code of RSCN-INFO is summarized in Algorithm 1.
The calculation complexity The computational complexity of the INFO algorithm depends on the size of the population N P , the times of iterations G max , and the dimensional search domain D. The complexity of INFO is OðN P Â G max Â DÞ. For SCN, assume that a set of datasets with N inputs X ¼ fx 1 ; x 2 ; …; x N g, and the maximum number of hidden layer neurons of SCN is L max . The main cost of SCN is caused by computing Moore-Penrose pseudo-inverse H y L T. A rough estimate of the computational complexity of H y L can be expressed as OðNL 3 max þ N 2 L 2 max þ L 4 max Þ (Li & Wang, 2017). Note that the cost of H y L T is calculated by

Algorithm 1 RSCN-INFO
Training dataset: ðx i ; t i Þ N i¼1 , x i 2 R d ; t i 2 R m . Parameters: the population size N P , maximum number of generations G max , dimensional search domain D, upper bounds ub and lower bounds lb of λ, r, η and γ, maximum number of hidden layer neurons L max , residual error threshold τ, maximum number of random assignment T max .

18:
Compute g L and n L;q by Eqs. (33) and (34) 19: if w Ã L and b Ã L satisfy constraint inequality then 20: Save the random parameters in W, n L in Ω; the widely-used singular value decomposition. Hence, the total complexity of RSCN-

EXPERIMENTS
The effectiveness of RSCN-INFO was evaluated on a function approximation and three benchmark datasets from the Knowledge Extraction based on Evolutionary Learning (KEEL, http://www.keel.es) dataset repository supported by the Spanish Ministry of Science and Technology. The approximation function is a conventional high nonlinear compound function that is widely used to evaluate randomized neural networks. The KEEL dataset contains classification, regression, unsupervised, and time series datasets. To verify the effectiveness of RSCN-INFO, it was compared with classical IRVFL, SCN (Wang & Li, 2017b), RSCN , and DASCN-II (Wang et al., 2020). All the experiments were implemented with MATLAB R2019b on a PC with AMD Ryzen 7 3.20 GHz CPU, NVIDIA GeForce MX450 GPU, and 16 GB RAM.

Benchmark datasets
Three real-world benchmark datasets for regression from KEEL were employed as experimental datasets. Specifications of these datasets are given in Table 2.
Figures 5-7 and Tables 3-5 depict the average training and test results on these benchmark datasets. Each test's statistical results for the 20 run times and the average value of RMSE were selected to evaluate the performance of the different algorithms. The IRVFL could not reach the preset tolerance threshold, so it was omitted here. In this case, the scope of random parameters λ in SCN was selected from the set f1 : 0:1 : 5g and the lower and upper bounds of λ in RSCN-INFO were set to ½1; 5. All the other parameters were set the same as the function approximation. Figure 5 shows similar performance between RSCN-INFO and the competitor algorithms on concrete. The reason for this phenomenon is that the concrete dataset contained smaller features and instances. For the compactiv and pole datasets, Figs. 6 and 7 clearly show that RSCN-INFO can achieve lower RMSE in terms of both training and test results. Intuitively and obviously, the RMSE of RSCN was used as the fitness function of INFO. In essence, INFO explored a global optimum solution that minimizes the fitness function in a four-dimensional search domain (λ; r; c; g) over several successive generations.   To verify the effectiveness of RSCN-INFO, Table 6 lists the computational time among SCN, RSCN, DASCN-II, and RSCN-INFO on benchmark datasets. We found that the training time of RSCN-INFO was significantly shorter than the other methods on the three benchmark datasets. Table 6 indicates that RSCN-INFO which employed optimized parameters and could achieve better efficiency. It should be noted that we did not take the parameter optimization process into account in this experiment. The optimization process   may consume additional time. However, the improvement of regression accuracy and network structure may be worth the time that is spent on the parameter optimization.
To further illustrate the network compactness of RSCN-INFO, we investigated how many hidden nodes were required to meet a preset error tolerance. As shown in Fig. 8, RSCN-INFO requires fewer hidden nodes compared with other methods. It can be deduced that given a preset s, RSCN-INFO can reach the error tolerance using fewer hidden neurons. This is due to RSCN-INFO using optimized parameters that can achieve a higher residual error reduction. Therefore, the network structure is more compact. It should be pointed out that DASCN-II can also construct a relatively compact SCN. However, the tunable value c in DASCN-II is a fixed value. It is selected empirically and difficult to adjust. Moreover, an inappropriate c will seriously affect the accuracy of the model.
In classical SCN and its various variants, λ tends to set a relatively larger value in complex problems. The parameter r is unfixed and set based on an increasing sequence from 0.9 to 1. The other parameters are selected empirically in connection with practical applications. Therefore, the conclusion may be drawn that RSCN-INFO is not only helpful in adaptively selecting parameters of SCN, but also beneficial for constructing a compact network. areas in multi-dimensional search space. A higher residual error decreasing rate is impacted by the parameter selection of RSCN-INFO. The experimental results on a function approximation and three benchmark regression datasets from KEEL indicated that the proposed RSCN-INFO algorithm exhibits considerable advantages in parameter optimization and network structure compactness compared with other algorithms. In almost all practical modeling tasks, the presence of noise and outliers is inevitable. This optimization strategy will accelerate the degradation of the learning performance of SCN that are subjected to noise or outliers. The robust skills used to weaken the negative influences of noise and outliers will be further discussed.